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SUMMARY. Introduction: XXII European 
Congress of Medical Informatics (MIE 2009) 
took place in Sarajevo from August 30th to 
September 2nd 2009. Assessment of qual- 
ity of papers presented at MIE 2009 was 
a process of observation, measurement, 
comparison and evaluation of the quality 
of orally presented papers. Methodology: 
For this study, and for the first time since 
EFMI founding (1976) and MIE congresses, 
the authors introduced a specially created 
quality assessment form with five relevant 
paper quality variables (methodological 
approach, international influence, scientific 
content, language quality, technical features) 
which the first author of this article used in 
peer-review process of papers submitted for 
publication in the journal Acta Informatica 
Medica (as Editor-in-Chief for last 1 8 years). 
The survey was conducted on the principle 



of random sampling of participants of MIE 
2009 Conference in Sarajevo, where specially 
trained interviewers (final year students of 
medicine and engineering at the University of 
Sarajevo) interviewed 33 session's chairs and 
1 1 0 participants/listeners of MIE 2009 paper 
presentations in 33 sessions (of total 40). 
Data was collected, entered into a specially 
created database, analyzed and presented. 
Results: From the total of 1 50 oral presenta- 
tions at the MIE 2009, 1 1 0 oral presentations 
were graded by both chairs and participants/ 
listeners. Grading results were compared and 
we found that in 60% of cases (66 papers) 
session chairs gave higher ratings than other 
participants of the congress. The highest rat- 
ing was 1 0, and the lowest 3. Only 3 of the 
papers received all four grades 10 from the 
session chairs. The most common grade given 
by chairs of the session was 8 (26.36%), fol- 



lowed by 7 (20%), 9 (19.32%), 6 (13.18%), 
10 and 5 (7.50%), 4 (5%) and 3 (1.14%). 
Significant differences in quality assessment 
of papers done by chairs and those done by 
other participants/listeners are observed. 
Conclusion: This work should demonstrate 
the importance of introducing universal 
(uniform) scale for assessment of articles at 
conferences that would provide objective and 
relevant assessment, which has not been the 
practice. Results obtained using a single stan- 
dardized scale can be compared to each other 
and thus improve the quality of the articles 
and the congress. Future congresses can be 
organized in this manner and become leading 
events in certain fields of medical science. 
Keywords: medical informatics, qual- 
ity assessment, paper presentation, 
evaluation 



1. INTRODUCTION 

Assessment of quality of papers 
presented at MIE 2009 Conference 
in Sarajevo (August 30th to Septem- 
ber 2nd 2009) was a process of ob- 
servation, measurement, comparison 
and evaluation of the quality of oral- 
ly presented papers (1, 2). Assess- 
ment may be affected by the follow- 
ing factors: the oratory skills (style 
and manner of expression) of the 
presenter, active knowledge or igno- 
rance of the official language (in this 
case English), previous experience of 
presenting at similar conferences and 



sessions, subjective impression of 
listeners/raters (same country, pro- 
fessionally close/distant topic) etc. 

It is assumed that the evalua- 
tion made by session chairs should 
be more accurate in comparison to 
the evaluation made by other ses- 
sion participants/listeners, as it is the 
practice to have experts from the sci- 
entific field covering the session top- 
ic as chairs. However, it can happen 
that among the session participants/ 
listeners there are those who are 
more closely and/or professionally 
related to the topic and are important 



for the quality assessment and evalu- 
ation of the paper presentation. 

2. METHODOLOGY 

During four working days of MIE 
2009 Conference there were 40 ses- 
sions and 259 papers were presented 
as oral presentations, poster presen- 
tations, tutorials or workshops. This 
study presents the result of quality 
assessment evaluation done by ses- 
sion chairs and other randomly cho- 
sen participants/listeners of papers 
presentations in 33 sessions. 

For this study, and for the first time 
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since EFMI founding (1976) and MIE 
congresses, the authors introduced a 
specially created quality assessment 
form with five relevant paper quality 
variables (methodological approach, 
international influence, scientific 
content, language quality, technical 
features), which the first author of 
this article used in peer-review pro- 
cess of papers submitted for publica- 
tion in the journal Acta Informatica 
Medica (as Editor-in-Chief for last 
18 years) (Figure 5). The survey was 
conducted on the principle of ran- 
dom sampling of participants of MIE 
2009 Conference in Sarajevo, where 
specially trained interviewers (final 
year students of medicine and engi- 
neering at the University of Saraje- 
vo) interviewed 33 session's chairs 
and 110 participants/listeners of MIE 
2009 paper presentations in 33 ses- 
sions (of total 40). Data was collect- 
ed, entered into a specially created 
database, analyzed and then present- 
ed in large number of different tables 
and charts (Figure 1). 

In this paper, we will present de- 
tailed results of the first and last ses- 
sion, and although this is a relative- 
ly small sample, it is methodologi- 
cally important for two reasons: first 
and last session presenters were re- 
nowned scientists and researchers in 
the field of medical informatics with 
a relatively large number of cited ref- 
erences in this area; thematic content 
of their lectures was covering glob- 
al, strategic and methodological fea- 
tures in the field of medical informat- 
ics. 

3. RESULTS 

We show detailed compara- 
tive evaluation from sessions where 
chairs were professionally closely re- 
lated to the topic and where present- 
ers were relatively younger experts 
in the field of medical informatics 
(Tables 1,2; Figures 2,3). In the ta- 
bles and charts presented, we can no- 
tice significant differences in quality 
assessment of papers done by chairs 
and those done by other participants/ 
listeners. 

Analysis of the evaluation also 
showed that in most cases results 
match in grading of the scientific 
content quality, although it was ex- 
pected that "stricter criteria" will be 



Chair's Assessment 


Track: A Oral Paper Session 1 National eHealth 

Chairs: Etienne De Clercq 


Scientific International Originality Technical Presentation 




Building eHealth National Strategies - The Romanian Experience 

Presenter: George I. MIHALAS 4 7 6 6 6 29 


Presenter: Etienne DE CLERCQ 


The Second Generation Slovenian Health Insurance Card 

Presenter: Anka BOLKA 4 6 6 7 6 29 


Presenter: Tomaz MARCUN 4 6 6 7 7 30 


Track: B Oral Paper Session 2 Social Networks and the Web 

Adnan Bajraktarevic 


Scientific International Originality Technical Presentation 


Sum: 


Presenter: Luis FERNANDEZ-LUQUE 9 9 10 10 10 48 


Presenter: Ugur BILGE 9 10 9 9 9 46 


Presenter: Syed Sibte Raza ABIDI 8 8 8 9 8 41 


Presenter: Michael RIGBY 10 10 10 8 9 47 


Track: C Oral Paper Session 3 Drugs and Patient Safety 

Dragica Milinkic 


Scientific International Originality Technical Presentation 


Sum: 


Drug Information Portal in Europe: Information Retrieval with Multiple Health Terminologies 

Presenter: Saoussen SAKJI 5 3 5 5 3 21 


Is the "International Classification for Patient Safety" a Classification? 

Presenter: Stefan SCHULZ 9 9 9 10 10 47 


Ontological Representation of Adverse Drug Reactions Using the Foundational Model of Anatomy 
20. oktobar 2009 
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Average Participant's Assessment 


Track: A Oral Paper Session 1 National eHealth 

Chairs: Etienne De Clercq 

Asim Kurjak No of Assessors: 


Avg Scientific Avg International Avg Originality Avg Presentation 


Avg Total: 


Building eHealth National Strategies - The Romanian Experience 


7,00 7,50 7,00 6,25 


27,75 


Primary Healthcare Research Network: The Belgian ResoPrim Recommendations 

Presenter: Etienne DE CLERCQ 4 


8,50 8,50 8,75 8,00 


33,75 




7,25 8,75 7,75 8,25 


32,00 


Presenter: Tomaz MARCUN 4 


7,75 9,00 8,75 8,75 


34,25 


Track: D Oral Paper Session 4 Bioinformatics 

Dusanka Boskovic No of Assessors: 


Avg Scientific Avg International Avg Originality Avg Presentation 


Avg Total: 


European Efforts in Nanoinformatics Research Applied to Nanomedicine 

Presenter: Stefano CHIESA 2 9,50 10,00 9,50 9,50 38,50 


Presenter: Kristina FISTER 3 8,33 7,67 8,33 7,33 31,67 


GeneMining: Identification, Visualization, and Interpretation of Brain Ageing Signatures 

Presenter: Sandra BRINGAY 4 


7,75 6,67 7,50 7,75 




Machine Learning Analysis of Proteomics Data for Early Diagnosis 

Presenter: Dmitry DEVETYAROV 1 5,00 6,00 6,00 6,00 23,00 


Cadiag-2 and Fuzzy Probability Logics 


Track: A Oral Paper Session S HIS 

Chairs: Pirkko Nykanen 

Zekerijah Sabanovic No of Assessors: 


Avg Scientific Avg International Avg Originality Avg Presentation 


Avg Total: 


Single Source Information Systems to Connect Patient Care and Clinical Research 

Presenter: Martin DUGAS 5 


4,60 5,00 4,80 6,20 


20,60 


22. oktobar 2009 







Figure 1. Sample of evaluation analysis results (sessions 1,2,3 - chairs; sessions 1,4,5 - other participants) 



applied by session chairs, since they 
are more referent professionals and 
experts in both Health Sciences and 
related Medical Informatics field. 

From the total of 150 oral pre- 
sentations at the MIE 2009, 110 
oral presentations were graded by 
both chairs and participants/listen- 
ers. Grading results were compared 
(as shown graphically in tables and 
graphs), and we found that in 60% 
of cases (66 papers) session chairs 
gave higher ratings than other par- 
ticipants of the congress. The high- 
est rating was 10, and the lowest 3. 
Only 3 of the papers received all four 
grades 10 from the session chairs 
(2 papers in the group 11 and one 
from the group 16). The most com- 
mon grade given by chairs of the ses- 
sion was 8 (26.36%), followed by 7 
(20%), 9 (19.32%), 6 (13.18%), 10 and 



5 (7.50%), 4 (5%) and 3 (1.14%) (Fig- 
ure 4). 

4. DISCUSSION 

One of the most respectable sci- 
entists, researchers and physicians 
in biomedical literature, which dealt 
with the concept of quality in gen- 
eral, especially in the field of medi- 
cine and health, was Avedis Donabe- 
dian, author of most quoted defini- 
tion of quality assessment: structure 
- process - outcome. In his fifty pub- 
lished articles in the field of quality 
he presents the modern approach to 
the transformation of thinking about 
health systems. Other authors, such 
as Louis, Lohr, Maxwel, Goves in 
their studies promote the importance 
of evaluation and assessment of qual- 
ity in medicine and healthcare. The 
author of this article is closely spe- 
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Oral Paper 


Scientific 


International 


Originality 


Presentation Total 




Session 1 


C.A. P.A. 


C.A. 


P.A. 


C.A. P.A. 


C.A. P.A. C.A. 


P.A. 


Mihalas 


4 7.00 


7 


7.50 


6 7.00 


b b.zb 15 


77 7C 

11 .Id 


Bolka 


4 7.25 


6 


8.75 


6 7.75 


b O.ZD 11 


77 nn 


Marcun 


4 7.75 


6 


9.00 


6 8.75 


1 o./j 13 


7/1 7C 


Table 1. Session 1: National eHealth. Chairs: Etienne De Clercq. C.A. 


- Chair's Assessment, P.A. 




Participant's Assessment 












Oral Paper 


Scientific 


International 


Originality 


Presentation Total 




Session 40 


C.A. P.A. 


C.A. 


P.A. 


C.A. P.A. 


C.A. P.A. C.A. 


P.A. 


Rognoni 


9 6.17 


9 


5.67 


9 5.50 


9 7.33 36 


24.67 


Hercigonja-Szekeres 9 4.11 


9 


4.22 


9 3.89 


9 5.11 36 


17.33 


Lessard 


9 7.44 


9 


6.56 


9 6.22 


9 5.44 36 


25.67 



Table 2. Session 40: Learning and education. Chairs: George I. Mihalas, C.A. 
Participant's Assessment 



■ Chair's Assessment, P.A. 




■UUirtAiiAiT^nl 
■ Pjhk iJ-n : A-;-: -: i>-w 



Figure 2. Total for Oral Paper Session 1 



I I 




Figure 3. Total for Oral Paper Session 40 




Figure 4. Percent of ratings given by the session's chairs 

cialized in the field of measuring the 
quality of medical education, as one 
segment in the field of quality health 
care, and in his research he uses a 
modified Lickert scale for measuring 
the quality of education (3, 4, 5, 6). 
Assessment can be prospective or 



retrospective (1, 2, 
3, 4). Quality con- 
trol is a synonym for 
quality assessment 
and is increasing- 
ly replaced by this 
term (5, 6). XXII 
European Congress 
of Medical Infor- 
matics (MIE 2009) 
took place in Saraje- 
vo from August 30th 
to September 2nd 
2009. Presenters of 
oral presentations 
were arranged in 40 
sessions and there 
were total 150 pre- 
sentations. Chairs of 
the group sessions 
and participants of 
the congress graded 
the quality of oral 
presentations. Ses- 
sion chairs grading 
was done in 5 cat- 
egories: scientific 
work, international- 
ly, support to tech- 
nical characteristics 
of work, originality 
(news), and presen- 
tations. Other con- 
gress participants 
evaluated papers in 
4 categories: scien- 
tific work, interna- 
tionally, originali- 
ty (news), and presentation. They did 
not grade the technical characteris- 
tics of the work. For these reasons, 
in this paper we compared grades 
for scientific work, international- 
ly, originality (news) and the pre- 
sentation which were given by both 




the session chairs and participants of 
the congress. All grades were sum- 
marized and compared. The result- 
ing data are presented in tables and 
charts (Tables 1,2; Figures 1.2,3,4). 

5. CONCLUSION 

The specificity of this work, and 
also we assume its value, is that on 
the basis of the available literature 
we did not find any article which 
compared the evaluation of oral pre- 
sentations by the session chairs and 
participants of the congress. 

This work should demonstrate the 
importance of introducing universal 
(uniform) scale for assessment of ar- 
ticles at conferences that would pro- 
vide objective and relevant assess- 
ment, which has not been the prac- 
tice. Results obtained using a single 
standardized scale can be compared 
to each other and thus improve the 
quality of the articles and the con- 
gress. Future congresses can be or- 
ganized in this manner and become 
leading events in certain fields of 
medical science. 

Based on the indicators from the 
processed survey data (survey forms 
for quality assessment as those used 
during the MIE 2009 Conference in 
Sarajevo) the following could be se- 
lected: 

• Articles which will be declared as 
the best and rewarded at the end 
of MIE Conference; 

• Certain number of articles for 
publication in indexed journals in 
the field of Medical Informatics 
of general educational interest to 
the general readership and which 
will carry the appropriate number 
of ECTS points for the MSc and 
PhD students. 

This kind of selection for the best 
quality of articles, presented at the 
congress, can be a sign and stimulus 
to future participants of MIE Confer- 
ences, if they have information that 
the quality assessment, as a specific 
method of assessment of articles, is 
used during the congresses of Med- 
ical Informatics at the international 
level. Such potential participants will 
be motivated to make better presen- 
tation of their research results, be- 
cause they expect it to be published 
in one of prestige journals of Medical 
Informatics. 



VOL 18 N0 1 MARCH 2010 



Quality Assesment of Mie 2009 Sarajevo Conference Presentations 



CHAIR 1:_ 
CHAIR 2: 



Dear colleague, 



Paper 1: 
Paper 2: 
Paper 3: 
Paper 4: 



Please scoie the article accoiding to the following features that can be scored from 1 to 10: 



Scientific and research Contents of paper 



International importance and influence of paper 



Originality of application and methodology and 
research realization 



Iff 



Iff 



WW 



Iff 



6 7 
Iff Iff 



Iff 



Iff 



Quality of Technical features 



Quality of English language 



The score of the article according to the above criteria is: 



points. 



Date 



Please give the scoring form to the MIE 
2009 staff (to be sent to the Speaker 
Ready Room "NERETVICA") 



Signature 



Figure 5. Quality assessment form for session chairs (based on Acta Medic Informatica peer-review form) 
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eHealth Week 2010 offers unprecedented opportunities to both 
attendees and exhibitors as it brings together, for the first time ever, 
the High level eHealth 201 0 Conference with the World of Health , 
IT (WoHIT) Conference and Exhibition. Assembling all sectors L 
of Europe's Healthcare IT community, eHealth Week 201 0 offers J' 
practical examples of how eHealth is transforming the European A 
Healthcare landscape. 
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